Sharper lower bounds on the performance of the empirical risk minimization algorithm
نویسندگان
چکیده
In this note we study lower bounds on the empirical minimization algorithm. To explain the basic set up of this algorithm, let (Ω, μ) be a probability space and set X to be a random variable taking values in Ω, distributed according to μ. We are interested in the function learning (noiseless) problem, in which one observes n independent random variables X1, . . . , Xn distributed according to μ, and the values T (X1), . . . , T (Xn) of an unknown target function T . The goal is to construct a procedure that uses the data D = (Xi, T (Xi))1≤i≤n with a risk as close as possible to the best one in F ; that is, we want to construct a statistic f̂n satisfying that for every n, with high μn-probability
منابع مشابه
Learning from uniformly ergodic Markov chains
Evaluation for generalization performance of learning algorithms has been themain thread of machine learning theoretical research. The previous bounds describing the generalization performance of the empirical risk minimization (ERM) algorithm are usually establishedbased on independent and identically distributed (i.i.d.) samples. In this paper we go far beyond this classical framework by esta...
متن کاملEmpirical Risk Minimization: Probabilistic Complexity and Stepsize Strategy
Empirical risk minimization (ERM) is recognized as a special form in standard convex optimization. When using a first order method, the Lipschitz constant of the empirical risk plays a crucial role in the convergence analysis and stepsize strategies for these problems. We derive the probabilistic bounds for such Lipschitz constants using random matrix theory. We show that, on average, the Lipsc...
متن کاملRanking and Scoring Using Empirical Risk Minimization
A general model is proposed for studying ranking problems. We investigate learning methods based on empirical minimization of the natural estimates of the ranking risk. The empirical estimates are of the form of a U -statistic. Inequalities from the theory of U -statistics and U processes are used to obtain performance bounds for the empirical risk minimizers. Convex risk minimization methods a...
متن کاملApplications of Empirical Processes in Learning Theory: Algorithmic Stability and Generalization Bounds
This thesis studies two key properties of learning algorithms: their generalization ability and their stability with respect to perturbations. To analyze these properties, we focus on concentration inequalities and tools from empirical process theory. We obtain theoretical results and demonstrate their applications to machine learning. First, we show how various notions of stability upperand lo...
متن کاملLearning with Noisy Labels
In this paper, we theoretically study the problem of binary classification in the presence of random classification noise — the learner, instead of seeing the true labels, sees labels that have independently been flipped with some small probability. Moreover, random label noise is class-conditional — the flip probability depends on the class. We provide two approaches to suitably modify any giv...
متن کامل